<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <title>External File support</title>
    <link rel="stylesheet" href="gettingStarted.css" type="text/css" />
    <meta name="generator" content="DocBook XSL Stylesheets V1.73.2" />
    <link rel="start" href="index.html" title="Berkeley DB Programmer's Reference Guide" />
    <link rel="up" href="am_misc.html" title="Chapter 4.  Access Method Wrapup" />
    <link rel="prev" href="am_misc_diskspace.html" title="Disk space requirements" />
    <link rel="next" href="am_misc_db_sql.html" title="Specifying a Berkeley DB schema using SQL DDL" />
  </head>
  <body>
    <div xmlns="" class="navheader">
      <div class="libver">
        <p>Library Version 12.1.6.2</p>
      </div>
      <table width="100%" summary="Navigation header">
        <tr>
          <th colspan="3" align="center">External File support</th>
        </tr>
        <tr>
          <td width="20%" align="left"><a accesskey="p" href="am_misc_diskspace.html">Prev</a> </td>
          <th width="60%" align="center">Chapter 4.  Access Method Wrapup </th>
          <td width="20%" align="right"> <a accesskey="n" href="am_misc_db_sql.html">Next</a></td>
        </tr>
      </table>
      <hr />
    </div>
    <div class="sect1" lang="en" xml:lang="en">
      <div class="titlepage">
        <div>
          <div>
            <h2 class="title" style="clear: both"><a id="blobs"></a>External File support</h2>
          </div>
        </div>
      </div>
      <div class="toc">
        <dl>
          <dt>
            <span class="sect2">
              <a href="blobs.html#blob_threshold">The external file threshold</a>
            </span>
          </dt>
          <dt>
            <span class="sect2">
              <a href="blobs.html#blob_create">Creating external files</a>
            </span>
          </dt>
          <dt>
            <span class="sect2">
              <a href="blobs.html#blob_access">External file access</a>
            </span>
          </dt>
          <dt>
            <span class="sect2">
              <a href="blobs.html#blob_storage">External file storage</a>
            </span>
          </dt>
          <dt>
            <span class="sect2">
              <a href="blobs.html#blob_replication">External Files and Replication</a>
            </span>
          </dt>
        </dl>
      </div>
      <p>
        External file support is designed for
        efficient storage of large objects. An object is considered to
        be large if it is more than a third of the size of a page.
        Without external file support, large objects must be broken up into
        smaller pieces, and then reassembled and/or disassembled every
        time the record is read or updated. Berkeley DB external file support
        avoids this assembly/disassembly process by storing the large
        object in a special directory set aside for the purpose. The
        data itself is not kept in the database, nor is it placed into
        the in-memory cache. 
    </p>
      <p> 
        external files can only be stored using the data portion of a
        key/data pair. They are supported only for Btree, Hash, and
        Heap databases, and only so long as the database is not
        configured for duplicate records, or
        duplicate sorted records. In addition, the DBT that you use to
        access the external file data cannot be configured as a partial DBT if
        you want to access the data using the external file's streaming
        interface (introduced below).
    </p>
      <p>
        Note that if the environment is transactionally-protected,
        then all access to the external file is also transactionally protected.
    </p>
      <div class="sect2" lang="en" xml:lang="en">
        <div class="titlepage">
          <div>
            <div>
              <h3 class="title"><a id="blob_threshold"></a>The external file threshold</h3>
            </div>
          </div>
        </div>
        <p>
            The external file threshold is a positive integer, in bytes,
            which indicates how large an object must be before it is
            considered an external file. By default, the external file
            threshold for any given database is <code class="literal">0</code>, which
            means that no object will ever be considered an external file. This
            means that the external file feature is not used by default for
            Berkeley DB databases.
        </p>
        <p>
            In order to use the external file feature, you must set the
            external file threshold to a non-zero, positive integer value. You
            do this for a given database using the <a href="../api_reference/C/set_blob_threshold.html" class="olink">DB-&gt;set_blob_threshold()</a>
            method. Note that this value must be set before you create
            the database. At any point after database creation time,
            this method is ignored. 
        </p>
        <p> 
            In addition, if you are using an environment, you can
            change the default threshold for databases created in that
            environment to something other than <code class="literal">0</code>
            by using the <a href="../api_reference/C/envset_blob_threshold.html" class="olink">DB_ENV-&gt;set_blob_threshold()</a> method. 
        </p>
        <p> You can retrieve the external file threshold set for a database
            using the <a href="../api_reference/C/get_blob_threshold.html" class="olink">DB-&gt;get_blob_threshold()</a>. You can retrieve the
            default external file threshold set for your environment using the
            <a href="../api_reference/C/envget_blob_threshold.html" class="olink">DB_ENV-&gt;get_blob_threshold()</a>.
        </p>
      </div>
      <div class="sect2" lang="en" xml:lang="en">
        <div class="titlepage">
          <div>
            <div>
              <h3 class="title"><a id="blob_create"></a>Creating external files</h3>
            </div>
          </div>
        </div>
        <p>
            There are two ways to create an external file. Before you can use
            either mechanism, you must set the external file threshold to a
            non-zero positive integer value (see the previous section
            for details). Once the external file threshold has been set, you
            create an external file using one of the two following mechanisms: 
        </p>
        <div class="itemizedlist">
          <ul type="disc">
            <li>
              <p>
                    Configure the DBT used to access the external file data
                    (that is, the DBT used for the data portion of the
                    record) with the <a href="../api_reference/C/dbt.html#dbt_DB_DBT_BLOB" class="olink">DB_DBT_BLOB</a> flag. This causes
                    the data to be stored as an external file regardless of its
                    size, so long as the database otherwise supports
                    external files.
                </p>
            </li>
            <li>
              <p>
                    Alternatively, creating a data item with a size
                    greater than the external file threshold will cause that
                    data item to be automatically stored as an external file.
                </p>
            </li>
          </ul>
        </div>
      </div>
      <div class="sect2" lang="en" xml:lang="en">
        <div class="titlepage">
          <div>
            <div>
              <h3 class="title"><a id="blob_access"></a>External file access</h3>
            </div>
          </div>
        </div>
        <p> 
            external files may be accessed in the same way as other DBT
            data, so long as the data itself will fit into memory.
            More likely, you will find it necessary to use the external file
            streaming API to read and write external file data. You open an
            external file stream using the <a href="../api_reference/C/dbstream.html" class="olink">DBC-&gt;db_stream()</a> method, close it with the
            <a href="../api_reference/C/dbstream_close.html" class="olink">DB_STREAM-&gt;close()</a> method, write to it using the the
            <a href="../api_reference/C/dbstream_write.html" class="olink">DB_STREAM-&gt;write()</a> method, and read it using the
            <a href="../api_reference/C/dbstream_read.html" class="olink">DB_STREAM-&gt;read()</a> method. 
        </p>
        <p> 
            The following example code fragment can be found in
            your DB distribution at
                <code class="filename">.../db/examples/c/ex_external_file.c</code>. 
        </p>
        <pre class="programlisting">...
    /* Some necessary variable declarations */
    DBC *dbc;       /* Cursor handle */
    DB_ENV *dbenv;  /* Environment handle */
    DB *dbp;        /* Database handle */
    DB_STREAM *dbs; /* Stream handle */
    DB_TXN *txn;    /* Transaction handle */
    DBT data, key;  /* DBT handles */
    int ret;
    db_off_t size;

    ...

    /* Environment creation skipped for brevity's sake */

    ...

    /* Enable external files and set the size threshold. */
    if ((ret = dbenv-&gt;set_ext_file_threshold(dbenv, 1000, 0)) != 0) {
        dbenv-&gt;err(dbenv, ret, "set_ext_file_threshold");
        goto err;
    }

    ...

    /* Database and DBT creation skipped for brevity's sake */

    ...

    /* 
        Access the external file using the DB_STREAM API. 
    */
    if ((ret = dbenv-&gt;txn_begin(dbenv, NULL, &amp;txn, 0)) != 0){
        dbenv-&gt;err(dbenv, ret, "txn");
        goto err;
    }

    if ((ret = dbp-&gt;cursor(dbp, txn, &amp;dbc, 0)) != 0) {
        dbenv-&gt;err(dbenv, ret, "cursor");
        goto err;
    }

    /*
     * Set the cursor to an external file.  Use DB_DBT_PARTIAL with
     * dlen == 0 to avoid getting any external file data.
     */
    data.flags = DB_DBT_USERMEM | DB_DBT_PARTIAL;
    data.dlen = 0;
    if ((ret = dbc-&gt;get(dbc, &amp;key, &amp;data, DB_FIRST)) != 0) {
        dbenv-&gt;err(dbenv, ret, "Not an external file");
        goto err;
    }
    data.flags = DB_DBT_USERMEM;

    /* Create a stream on the external file the cursor points to.  */
    if ((ret = dbc-&gt;db_stream(dbc, &amp;dbs, DB_STREAM_WRITE)) != 0) {
        dbenv-&gt;err(dbenv, 0, "Creating stream.");
        goto err;
    }

    /* Get the size of the external file.  */
    if ((ret = dbs-&gt;size(dbs, &amp;size, 0)) != 0) {
        dbenv-&gt;err(dbenv, 0, "Stream size.");
        goto err;
    }
    /* Read from the external file. */
    if ((ret = dbs-&gt;read(dbs, &amp;data, 0, (u_int32_t)size, 0)) != 0) {
        dbenv-&gt;err(dbenv, 0, "Stream read.");
        goto err;
    }
    /* Write data to the external file, increasing its size. */
    if ((ret = dbs-&gt;write(dbs, &amp;data, size/2, 0)) != 0) {
        dbenv-&gt;err(dbenv, 0, "Stream write.");
        goto err;
    }
    /* Close the stream. */
    if ((ret = dbs-&gt;close(dbs, 0)) != 0) {
        dbenv-&gt;err(dbenv, 0, "Stream close.");
        goto err;
    }
    dbs = NULL;
    dbc-&gt;close(dbc);
    dbc = NULL;
    txn-&gt;commit(txn, 0);
    txn = NULL;
    free(data.data);
    data.data = NULL; 

    ...

    /* Handle clean up skipped. */ </pre>
      </div>
      <div class="sect2" lang="en" xml:lang="en">
        <div class="titlepage">
          <div>
            <div>
              <h3 class="title"><a id="blob_storage"></a>External file storage</h3>
            </div>
          </div>
        </div>
        <p>
            external files are not stored in the normal database files on
            disk in the same way as is other data managed by DB.
            Instead, they are stored as binary files in a special
            directory set aside for the purpose.
        </p>
        <p>
            If you are not using environments, this special external file
            directory is created relative to the current working
            directory from which your application is running. You can
            modify this default location using the <a href="../api_reference/C/set_blob_dir.html" class="olink">DB-&gt;set_blob_dir()</a>
            method, and retrieve the current external file directory using
            <a href="../api_reference/C/get_blob_dir.html" class="olink">DB-&gt;get_blob_dir()</a>.
        </p>
        <p>
            If you are using an environment, then by default the
            external file directory is created within the environment's home
            directory. You can change this default location using
            <a href="../api_reference/C/envset_blob_dir.html" class="olink">DB_ENV-&gt;set_blob_dir()</a> and retrieve the current default
            location using <a href="../api_reference/C/envget_blob_dir.html" class="olink">DB_ENV-&gt;get_blob_dir()</a>. (Note that
            <a href="../api_reference/C/envget_blob_dir.html" class="olink">DB_ENV-&gt;get_blob_dir()</a> can successfully retrieve the external file
            directory only if <a href="../api_reference/C/envset_blob_dir.html" class="olink">DB_ENV-&gt;set_blob_dir()</a> was previously
            called.)
        </p>
        <p> 
            Note that because external files are stored outside of the
            Berkeley DB database files, they are not confined by the
            four gigabyte limit used for Berkeley DB key and data
            items. The external file size limit is system dependent. It can be
            the maximum value in bytes of a signed 32 bit integer (if
            the Berkeley DB-defined type <code class="literal">db_off_t</code>
            is four bytes in size), or a signed 64 bit integer (if
            <code class="literal">db_off_t</code> is eight bytes in size).
        </p>
      </div>
      <div class="sect2" lang="en" xml:lang="en">
        <div class="titlepage">
          <div>
            <div>
              <h3 class="title"><a id="blob_replication"></a>External Files and Replication</h3>
            </div>
          </div>
        </div>
        <p>
            Replication supports external files without any special
            requirements.  However, enabling external files in a replicated
            environment can result in long synchronization times between the
            client and master sites.  To avoid this, execute a transaction
            checkpoint after updating or deleting one or more external file
            records.
        </p>
      </div>
    </div>
    <div class="navfooter">
      <hr />
      <table width="100%" summary="Navigation footer">
        <tr>
          <td width="40%" align="left"><a accesskey="p" href="am_misc_diskspace.html">Prev</a> </td>
          <td width="20%" align="center">
            <a accesskey="u" href="am_misc.html">Up</a>
          </td>
          <td width="40%" align="right"> <a accesskey="n" href="am_misc_db_sql.html">Next</a></td>
        </tr>
        <tr>
          <td width="40%" align="left" valign="top">Disk space
        requirements </td>
          <td width="20%" align="center">
            <a accesskey="h" href="index.html">Home</a>
          </td>
          <td width="40%" align="right" valign="top"> Specifying a Berkeley DB schema
        using SQL DDL</td>
        </tr>
      </table>
    </div>
  </body>
</html>
